Image processing apparatus, image distribution system, and image processing method

ABSTRACT

In an image processing apparatus, a region extraction part extracts an image of the sky as an image of a target object not included in an object of interest in a moving image of content, on the basis of at least any of changes in pixel values, a frequency at which a line of sight is directed, a distance from a camera, registered information designating the target object, and the like. In the extracted region, a different image presentation part displays an image indicative of a magnified image of a player and character information for cheering as an image different from the sky image.

TECHNICAL FIELD

The present invention relates to an image processing apparatus forprocessing content involving image display, an image distributionsystem, and an image processing method.

BACKGROUND ART

Advances in image processing technology and the expansion of the networkenvironment are making it possible to enjoy electronic content such asvideos and games casually in diverse forms. For example, a head-mounteddisplay worn by a user displays panoramic video image in a field of viewcorresponding to the face direction of the user. This enables enhancingthe sense of immersion in a visual world and improving the operabilityof applications such as games. Also, diverse wearable display devicespermit composition of a virtual object in a suitable position inreal-time video image captured in the user's field of view or in anappropriate position in a real image transmitted through a glass part.This makes it possible to merge the real world with the virtual world.

SUMMARY Technical Problem

It may be desired to let users enjoying the above-mentioned contentrecognize various kinds of notices and advertisements different from thedetails of the content. In such cases, an image carrying suchinformation may conceivably be brought into the user's field of view.However, the content being partially hidden by such an additional imagemay incur feelings of discomfort or botheration. Although the additionalimage might be displayed in a separate region within the screen, thisrequires shifting the line of sight far from the content; the additionalimage may thus fail to draw attention.

The present invention has been made in view of the above circumstances.An object of the invention is therefore to provide a technology by whichinformation is suitably presented in content including image display.

Solution to Problem

One embodiment of the present invention relates to an image processingapparatus. This image processing apparatus includes an image acquisitionpart configured to acquire data of a moving image representing a spaceto be viewed, a region extraction part configured to extract, as aregion of non-interest, a region of an image of any one of targetobjects not included in an object of interest, in a frame of the movingimage, and a different image presentation part configured to present andoutput an image different from the image of the target object in theregion of non-interest.

Another embodiment of the present invention relates to an imagedistribution system. This image distribution system includes theabove-described image processing apparatus, and a server configured totransmit the data of the moving image to the image processing apparatus,the server including a viewing rate calculation part configured toacquire information regarding a direction of a line of sight of a userviewing an image output from the image processing apparatus, the viewingrate calculation part further calculating a viewing rate of thedifferent image on the basis of changes in the direction of the line ofsight before and after the different image is displayed.

A further embodiment of the present invention relates to an imagedistribution system. This image distribution system includes a serverand an image processing apparatus. The server includes an imageacquisition part configured to acquire data of a moving imagerepresenting a space to be viewed, a region extraction part configuredto extract, as a region of non-interest, a region of an image of any oneof target objects not included in an object of interest in a frame ofthe moving image, and a different image presentation part configured topresent an image different from the image of the target object in theregion of non-interest. The image processing apparatus acquires over anetwork the data of the moving image output from the server andrepresenting the different image, and the image processing apparatusfurther causes a display apparatus to display the moving image.

A still further embodiment of the present invention relates to an imageprocessing method. This image processing method performed by an imageprocessing apparatus, includes a step of acquiring data of a movingimage representing a space to be viewed, a step of extracting, as aregion of non-interest, a region of an image of any one of targetobjects not included in an object of interest in a frame of the movingimage, and a step of outputting and presenting an image different fromthe image of the target object in the region of non-interest.

It is to be noted that arbitrary combinations of the above constituentelements as well as modes realized by converting expressions of thepresent invention between a method, a device, a system, a computerprogram, a recording medium on which the computer program is recorded,and the like are also effective as modes of the present invention.

Advantageous Effects of Invention

According to the present invention, information is suitably presented incontent including image display.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram depicting an example of an appearance of ahead-mounted display in the present embodiment.

FIG. 2 is a diagram depicting an example of a configuration of a contentprocessing system to which the present embodiment of this invention maybe applied.

FIG. 3 is a diagram depicting typical images of content to be processedby the present embodiment.

FIG. 4 is a diagram depicting an internal circuit configuration of animage processing apparatus in the present embodiment.

FIG. 5 is a diagram depicting functional block configurations of theimage processing apparatus and a server in the present embodiment.

FIG. 6 is a diagram explanatory of a timing for a different imagepresentation part in the present embodiment of this invention to presenta different image in a region of non-interest.

FIG. 7 is a diagram schematically depicting an example of an imagedisplayed in the present embodiment.

FIG. 8 is a diagram explanatory of how to obtain a viewing rate at atime of displaying a different image on AR glasses worn by audienceduring a concert in the real world, in the present embodiment.

FIG. 9 is a flowchart indicative of processing steps of displaying acontent image, the steps being performed by the image processingapparatus in the present embodiment.

DESCRIPTION OF EMBODIMENTS

The present embodiment relates to a technology for processing contentincluding image display. What details the image represents, what purposethe display is for, whether the video image is given in real time orrecorded, whether or not the display includes a real image, what formthe display device is in, or the like, are not particularly limited tothat extent. What follows is primarily an explanation of arepresentative example in which a moving image is displayed on ahead-mounted display. FIG. 1 is a diagram depicting an example of anappearance of a head-mounted display in the present embodiment. In thisexample, a head-mounted display 100 is configured with an outputmechanism part 102 and a wearing mechanism part 104. The wearingmechanism part 104 includes a wearing band 106 that, when worn by theuser, surrounds his or her head to secure the apparatus thereto.

The output mechanism part 102 includes a housing 108 shaped in such amanner as to cover both eyes when the head-mounted display 100 is wornby the user. Inside the housing 108, included is a display panel thatdirectly faces the eyes when the head-mounted display 100 is worn. Alsoprovided inside the housing 108 may be lenses interposed between thedisplay panel and the user's eyes when the head-mounted display 100 isworn, the lenses serving to magnify images. Preferably, a stereoscopicview may be implemented by displaying respective stereoscopic imagescorresponding to the parallax between the eyes in a right region and aleft region formed by bisecting the display panel.

The head-mounted display 100 may be further provided with speakers orearphones at positions corresponding to the user's ears upon wearing thehead-mounted display 100. In this example, the head-mounted display 100has a stereo camera 110 provided at the front of the housing 108 tocapture moving image of the surroundings in a field of viewcorresponding to the user's line of sight. Further, the head-mounteddisplay 100 may include, in or outside the housing 108, any one of suchdiverse sensors as an acceleration sensor, a gyro sensor, and ageomagnetic sensor for deriving a motion, a posture, and a position ofthe head-mounted display 100.

FIG. 2 is a diagram depicting an example of a configuration of a contentprocessing system to which the present embodiment can be applied. Thehead-mounted display 100 is connected by wireless communication to animage processing apparatus 10. Alternatively, the connection may be awired connection using a USB (Universal Serial Bus) cable, for example.The image processing apparatus 10 is connected with a server 200 via anetwork 8. The server 200 transmits moving images and sounds of adisplay object as content data to the image processing apparatus 10.

The image processing apparatus 10 performs necessary processes on thecontent data transmitted from the server 200, and sends the processeddata to the head-mounted display 100. Alternatively, the imageprocessing apparatus 10 may process internally-held content data togenerate the image and sound data or perform predetermined processes onthe moving image being captured by the stereo camera 110 of thehead-mounted display 100, before transmitting the processed data to thehead-mounted display 100. In such cases, the server 200 may not beneeded. As another alternative, the image processing apparatus 10 may beincorporated in the head-mounted display 100.

In addition, as mentioned above, the device for displaying moving imagesin this embodiment is not limited to the head-mounted display. Thedisplay device may alternatively be any one of diverse wearabledisplays, a flat-screen display such as a television receiver, a mobileterminal, or a projector. In a case where the head-mounted display 100is adopted, the image processing apparatus 10 acquires continuously theposition and posture of the user wearing the head-mounted display 100based on measured values from motion sensors incorporated in thehead-mounted display 100 or on images captured by the stereo camera 11,and generates a display image in a field of view corresponding to them.

A representative example of such display technology described above isvirtual reality (VR) that presents the image corresponding to the user'sfield of view in a three-dimensional space indicative of a virtual worldor in a previously acquired panoramic image. Also, the image processingapparatus 10 may implement augmented reality (AR) by rendering a virtualobject in a suitable position of the image being captured in real timeby the stereo camera 110. Alternatively, the image processing apparatus10 may reproduce the content image in a fixed field of view, regardlessof a motion of the user's head.

FIG. 3 is a diagram depicting typical images of content to be processedin the present embodiment. In this example, images of a baseball game ina baseball stadium are assumed. For instance, if the entire stadiumduring the game is captured with a wide-angle lens, and the image suchas one in Subfigure (a) is displayed in a field of view corresponding tothe head motion of the user wearing the head-mounted display 100, even auser at a remote location can experience realistic sensations as if theuser was also viewing the game from the stands. Note that, if similarimages are displayed as stereoscopic images for the right and left eyeswith a parallax therebetween, a stereoscopic view can be implemented.Such images may also be obtained by the stereo camera 110 of thehead-mounted display 100 worn by a user actually in the stands.

In a case where the image in Subfigure (a) is displayed, an objectdrawing the user's attention is primarily a field on which the playersare present. There is a low possibility that a sky image 120 or the likewill draw attention, for example. This embodiment thus utilizes such aregion out of the object of interest to present various images andvarious pieces of information. That is, a region of an image of any oneof target objects not included in the object of interest is extracted asa “region of non-interest” and an image different from the target objectis displayed.

In the illustrated example, as depicted in Subfigure (b), an image 122is displayed in the sky image region to indicate a magnified image of aplayer and character information for cheering. The object to bedisplayed may be not only what is present in the same space, but alsoany of such diverse objects as an image captured in a different space,notices for the user, advertisements, or a comment from another userwatching the same video image. The object to be displayed may be amoving image, a still image, character information, or a combinationthereof. In the description that follows, the object as the region ofnon-interest is referred to as the “replacement object,” and an imagedifferent from the image of the replacement object to be displayed inthe region of non-interest is referred to as the “different image.”

Suitably selecting the region of non-interest permits visual recognitionof necessary information without hiding the object of interest whileminimizing the movement of the line of sight. Further, replacing theimage of the actually-captured object with a different image bringsabout a sense of unity between the original video image and thedifferent image. Especially in a case where the field of view is variedin keeping with the head motion of the user wearing the head-mounteddisplay 100, the unity between the different image and the remainingimage reduces the feelings of discomfort upon changing of the field ofview and is not likely to impair realistic sensations.

FIG. 4 is a diagram depicting an internal circuit configuration of theimage processing apparatus 10. The image processing apparatus 10includes a CPU (Central Processing Unit) 23, a GPU (Graphics ProcessingUnit) 24, and a main memory 26. These components are interconnected viaa bus 30. The bus 30 is further connected with an input/output interface28. The input/output interface 28 is connected with a communication part32 constituted by a peripheral device interface supporting the USB andIEEE (Institute of Electrical and Electronics Engineers) 1394 standardsand by a wired or wireless LAN (Local Area Network) network interfacefor establishing communication with the server 200 and head-mounteddisplay 100, a storage part 34 such as a hard disk drive or anonvolatile memory, an output part 36 that outputs data to thehead-mounted display 100, an input part 38 that receives input of datafrom the head-mounted display 100, and a recording medium driving part40 that drives removable recording media such as a magnetic disk, anoptical disk, or a semiconductor memory.

The CPU 23 controls the image processing apparatus 10 as a whole byexecuting an operating system stored in the storage part 34. The CPU 23further executes various types of programs that are read out from theremovable recording medium and loaded into the main memory 26 or whichare downloaded via the communication part 32. The GPU 24 has thefunctions of both a geometry engine and a rendering processor. The GPU24 performs rendering processes according to rendering instructions fromthe CPU 23 and outputs the result of the processing to the output part36. The main memory 26 includes a RAM (Random Access Memory) that storesprograms and data necessary for the processing. Note that the server 200may also adopt the similar configuration as above.

FIG. 5 is a diagram depicting functional block configurations of theimage processing apparatus 10 and the server 200. The functional blocksillustrated in this drawing can be implemented by hardware such as theCPU 23, the GPU 24, and the main memory 26 depicted in FIG. 5 or bysoftware such as programs that are loaded from the recording medium intothe memory to implement such functions as information processing, imagerendering, data input/output, and communication. Thus, it will beunderstood by those skilled in the art that these functional blocks canbe implemented by hardware alone, by software alone, or by a combinationof both in diverse forms and that the implementation is not limited toany one of them.

The image processing apparatus 10 includes an image acquisition part 60that acquires data of a moving image indicative of a space to be viewed,a region extraction part 62 that extracts, from a frame of the movingimage, a region of an image of a replacement object as a region ofnon-interest, a different image presentation part 66 that displays adifferent image in the region of non-interest, a position informationacquisition part 64 that acquires a position of a subject in athree-dimensional image capturing space, a different image data storagepart 68 that stores data of the different image, a line-of-sightinformation acquisition part 72 that acquires information regarding adirection of a user's line of sight, and an output part 70 that outputsdata of a display image to the head-mounted display 100.

The image acquisition part 60 acquires from the server 200 data ofcontent selected by the user, for example. Alternatively, the imageacquisition part 60 may read out the content data from a storage deviceinside the image processing apparatus 10. Then, the image acquisitionpart 60 decompresses the data of the acquired image as needed, andsupplies the decompressed data to at least the region extraction part 62and the different image presentation part 66. Further, the imageacquisition part 60 may acquire from the server 200 what is needed outof the information necessary for extracting the region of non-interestor the information designating the region of non-interest, the positioninformation regarding the subject being captured in a moving image inthe three-dimensional space, and the data of a different image to bedisplayed or the information for designating the different image.

Given each frame of the moving image decompressed by the imageacquisition part 60, the region extraction part 62 extracts the imageregion of the replacement object as the region of non-interest. Forexample, the region extraction part 62 extracts the image that conformsto the following conditions:

-   -   1. An image of a target object of which constituent pixels have        their pixel values subjected to temporal changes equal to or        less than a predetermined value;    -   2. An image of a target object that tends not to draw much        attention in general;    -   3. An image of a target object to which the user's line of sight        is directed with a frequency equal to or less than a        predetermined value; and    -   4. An image of a target object to which a distance from the        camera capturing a moving image in a three-dimensional image        capturing space is equal to or more than a predetermined value.

Threshold values for use as criteria for the first, the third, and thefourth conditions above may be set for each moving image or shared forall moving images. For the second condition, the object that tends notto draw much attention in general, such as the sky or the ceiling, isdecided as the replacement object. Such information necessary forextracting the image as a name, a color, a shape, and feature pointdistribution of that object are registered in advance for each movingimage or in common to all moving images. In this case, the regionextraction part 62 extracts the image of the registered replacementobject, using existing target object recognition techniques such aspattern matching, feature point comparison, and background differencing.

For example, the sky image indicated in FIG. 3 applies to any of thefirst through the fourth conditions above. However, the conditions forextracting the region of non-interest are not intended to be limited tothe above conditions. For example, the region extraction part 62 maydetect an image motion based on frame-to-frame differences to extractthe region of an image deemed to manifest no movement. In some cases,the region extraction part 62 may acquire registered informationdesignating the region of non-interest itself from the server 200 or thelike, and supply the acquired information simply to the different imagepresentation part 66. Also, the region extraction part 62 may extract aregion that meets one condition or a region that meets two or moreconditions.

Note that, since common images are configured such that the object ofinterest is often concentrated from the center downward, the aboveconditions may be supplemented with another condition requiring thatgiven an upper region and a lower region vertically bisecting a frame ofthe moving image at a predetermined ratio, the upper region be selectedfor extraction of the region of non-interest. In any case, the regionextraction part 62 extracts the region of non-interest preferably alongthe contour of the replacement object in units of an image of thesubject. This makes it possible to bring about a sense of unity with theremaining subjects and not to impair realistic sensations as discussedabove.

Note that the region extraction part 62 may extract multiple regions ofnon-interest or move a region of interest according to changes in aconfiguration of the moving image or changes in the display region.Further, the region extraction part 62 also continuously monitors analready extracted region of non-interest to determine whether the regiondeviates from the condition for the extraction. If the extracted regionof non-interest is determined to have deviated from the condition, theregion extraction part 62 eliminates the region of non-interest. Whenthe region again meets the condition, the region extraction part 62 mayagain extract the same region as the region of non-interest.

Given each frame of the moving image, the different image presentationpart 66 generates a frame of the display image by composing a differentimage in the region of non-interest. As with the sky image depicted inFIG. 3 , the different image presentation part 66 may replace the entireregion of non-interest with the different image or place the differentimage only in a portion of the region. Also, in a case where characterinformation is presented as the different image, the different imagepresentation part 66 may display the characters along with thebackground such as a filled image or place only the characters directlyonto the image of the replacement object.

In a display form in which the field of view changes in keeping with thehead motion of the user, the different image presentation part 66 maymove the display position of the different image in a manner keepingtrack of the changing of the field of view. In the example of the imagesin FIG. 3 , for example, when the user turns to the right, the skyregion within the field of view moves. At this time, if the displayposition of the different image is shifted to keep track of the movementof the region, the different image can be continuously displayed in thesky region within the field of view. Alternatively, the different image,once displayed, may be continuously displayed in the same position inthe image of the replacement object, regardless of change in field ofview. In this case, the different image can be made to appear pasted onthe replacement object.

The different image presentation part 66 may read out the data of thedifferent image from the different image data storage part 68 or acquirethe data directly from the server 200. Also in the former case, theinformation identifying the data of the different image to be displayedand the information such as a display timing may also be acquired fromthe server 200. General computer graphics techniques can be used incomposing the different images. Also, the different image presentationpart 66 may set at least any of a frame rate or a resolution of thedifferent image to be lower than that of the moving image at thedestination for the composition. This reduces a data size of the displayimage.

The position information acquisition part 64 acquires positioninformation of the subject being presented in the moving image, in athree-dimensional image capturing space. The position informationacquisition part 64 may acquire the position information together withthe content data from the server 200. Alternatively, the positioninformation acquisition part 64 may acquire the position information byanalyzing the image of the content or the data of a depth image sentalong with the content image from the server 200. The depth image is acommon image that represents, in image analysis, the distance from thecamera to the subject in terms of the pixel values of the image.

The position information acquisition part 64 supplies the acquiredposition information to at least either the region extraction part 62 orthe different image presentation part 66. The region extraction part 62uses the position information when extracting as the region ofnon-interest an image meeting the fourth condition above, i.e., an imageof the object to which the distance from the camera capturing a movingimage in a three-dimensional image capturing space is equal to or morethan a predetermined value. In the case of an image presenting anoutdoor event, for example, the subject deemed sufficiently distantcompared with a size of a venue is highly likely not to be the object ofinterest, such as the sky or the mountains constituting the backgroundin the image.

In the case of an indoor event, too, the subject not likely to drawattention such as a ceiling or a wall is often far away from the cameracapturing the main object. Thus, by setting an appropriate thresholdvalue for the distance from the camera according to a scale of the spaceto be captured, it is possible suitably to identify the subject notlikely to draw attention. On the other hand, when composing thedifferent image in the extracted region of non-interest, the differentimage presentation part 66 decides a magnification factor for displayingthe different image on the basis of the distance from the camera to thereplacement object. Specifically, the more distant the replacementobject is, the lower the magnification factor for the different image tobe displayed thereon.

In this manner, the different image presentation part 66 can express thedifferent image with the same sense of distance as the replacementobject, allowing the different image to be recognized as closely a partof the image capturing space. In a case where stereoscopic view isimplemented using stereoscopic images with a parallax therebetween, thedifferent image presentation part 66 provides the parallax in a mannersimilar to that of the replacement object, thereby causing the differentimage to appear in the same position. Note that, if another condition isused for extracting the region of non-interest or if the magnificationfactor is fixed for displaying the different image, then the function ofthe position information acquisition part 64 may be omitted.

The line-of-sight information acquisition part 72 acquires informationregarding the direction of the user's line of sight at a predeterminedrate. For example, the line-of-sight information acquisition part 72acquires the head motion of the user from measured values of a motionsensor, not depicted, incorporated in the head-mounted display 100, andtakes the direction of a normal line to the user's face to be thelight-of-sight direction. This embodiment may adopt any of the variouscurrently available techniques for acquiring the head motion of the userwearing a wearable display such as the head-mounted display 100 andfurther, the face direction of the user.

Alternatively, the line-of-sight information acquisition part 72 mayacquire a position that actually draws attention in the display image onthe basis of information from a gaze point detector in the head-mounteddisplay 100. There are known techniques for emitting a reference beamsuch as infrared rays to the user's eyeball and acquiring the directionof the pupil as well as the gaze point based on the distribution of thereflected light. Any of the currently available techniques forimplementing the gaze point detector other than the above may beadopted.

The line-of-sight information acquisition part 72 supplies the acquiredinformation regarding the line of sight to one of the region extractionpart 62 and the server 200, or to both as needed. The region extractionpart 62 uses the line-of-sight information when extracting as the regionof non-interest the image of the subject to which the user's line ofsight is directed with a frequency equal to or less than a predeterminedvalue. For example, the region extraction part 62 extracts as the regionof non-interest the image of the object to which the line of sight hasbeen directed in an accumulated time period shorter than a predeterminedvalue most recently. In this manner, the “frequency at which the line ofsight is directed” may be a ratio of the time in which the line of sighthas been directed.

Alternatively, the region extraction part 62 may extract the image ofthe subject to which the line of sight has been directed as frequentlyas, or less frequently than, a threshold value over a predetermined timeperiod most recently. The server 200 uses line-of-sight information forextracting the region of non-interest and calculating the viewing rateof the different image as described later. It is to be noted, however,that this embodiment is not limited to using the line-of-sightinformation in extracting the region of non-interest or calculating theviewing rate. In a case where there is no need for the line-of-sightinformation, the function of the visual recognition informationacquisition part 2 can be omitted.

The output part 70 outputs the display image data generated by thedifferent image presentation part 66 to the head-mounted display 100 ata predetermined rate. Note that, whereas the output part 70 mayconcurrently output audio data included in the content data, theaudio-related processing is omitted from the drawing.

The server 200 includes a distribution processing part 50 thatdistributes content data, a content data storage part 52 that stores thecontent data, a region information acquisition part 54 that acquiresinformation regarding regions of non-interest, and a viewing ratecalculation part 56 that calculates the viewing rate of the differentimage. Upon request or the like from the image processing apparatus 10,the distribution processing part 50 reads out content data from thecontent data storage part 52 and transmits the data. Alternatively, thedistribution processing part 50 may transmit real-time video imagescaptured by a camera, not depicted, on the spot.

The region information acquisition part 54 acquires the informationrequired at a time of extracting the region of non-interest in the imageprocessing apparatus 10. For example, the region information acquisitionpart 54 acquires identification information identifying the replacementobjects decided in advance and their features, parameters for use inextracting the regions of non-interest, threshold values for use indetermining the regions, and the like. Such information may be preparedbeforehand in the content data storage part 52, read out therefrom andacquired. Alternatively, the information may be acquired by the regioninformation acquisition part 54 itself analyzing the moving image.Particularly in the case of recorded content, detailed informationregarding the content can be prepared through image analysis over longhours.

As another alternative, the region information acquisition part 54 mayextract the region of non-interest itself, using the acquiredinformation. The region information acquisition part 54 may furtheracquire the position of the subject in the three-dimensional imagecapturing space. In the case of distributing a video image captured inreal time, the region information acquisition part 54 extracts theregion of non-interest and acquires the position information regardingthe subject in units of a frame of the captured moving image or atpredetermined time intervals. In the case of extracting the region ofnon-interest, the region information acquisition part 54 uses techniquessimilar to those used by the region extraction part 62 in the imageprocessing apparats 10.

In a case where the replacement object is decided according to theactual line of sight of the user on the image processing apparatus 10 asper the fourth condition above, the region information acquisition part54 acquires the line-of-sight information obtained by the line-of-sightinformation acquisition part 72 in the image processing apparatus 10. Inthe case of obtaining the position of the subject in an image capturingspace, the region information acquisition part 54 acquires the distanceto the subject by the principle of triangulation, using stereoscopicimages captured of the image capturing space by a pair of cameraspositioned right and left with a predetermined distance apart, forexample.

Alternatively, the region information acquisition part 54 acquires thedistance to the subject by emitting a reference beam such as infraredrays to the subject and by measuring the time it takes to detect thereflected light therefrom. Such distance measurement techniques are wellknown. Note that a portion of these processes may be taken over by theposition information acquisition part 64 in the image processingapparatus 10. For example, the position of the subject may beapproximately estimated on the side of the server 200 and then acquiredat a higher resolution on the side of the image processing apparatus 10.Alternatively, the region of non-interest and the position of thesubject in the three-dimensional space may be acquired in detail on theside of the server 200, with the processing of the region extractionpart 62 or of the position information acquisition part 64 omitted onthe side of the image processing apparatus 10.

The region information acquisition part 54 may further read out from thecontent data storage part 52 the data of the different image to bedisplayed in the region of non-interest and the display-related rulessuch as the display timing of the different image and the displaymagnification factor thereof. As another alternative, the regioninformation acquisition part 54 may decide the display magnificationfactor of the different image in accordance with the distance of thereplacement object from the camera. The region information acquisitionpart 54 supplies the acquired information to the distribution processingpart 50 as needed.

Accordingly, the distribution processing part 50 transmits to the imageprocessing apparatus 10 not only the content data but also theinformation acquired by the region information acquisition part 54 andneeded for extracting the region of non-interest and for displaying thedifferent image. In this case, the distribution processing part 50 mayreduce the size of the data to be transmitted by downsizing an amount ofinformation by deleting, from the moving image, the image of thereplacement object extracted by the region information acquisition part54 as the region of non-interest and by increasing a compression ratioof that region, for example.

The viewing rate calculation part 56 acquires changes in the directionof the user's line of sight before and after the different image isdisplayed to estimate the viewing rate of the different image on thebasis of the acquired changes. For example, with a state prior todisplay of the different image taken as the reference, the viewing ratecalculation part 56 calculates as the viewing rate the frequency atwhich the line of sight is directed to the replacement object after thedifferent image is displayed. Alternatively, an amount of increase inthe frequency at which the line of sight is directed to the replacementobject following display of the different image may be taken as theviewing rate.

The frequency at which the line of sight is directed may be theaccumulated time for which the line of sight is directed in apredetermined time period or a ratio thereof as discussed above.Alternatively, the frequency may be the number of times the line ofsight has been directed over a predetermined time period. Note that, ina case where the timing for displaying the different image is notacquired on the side of the server 200, the line-of-sight informationacquisition part 72 in the image processing apparatus 10 acquires thattiming from the region extraction part 62 and notifies the server 200thereof. The viewing rate serves as an index indicative of the effectsof the different image being displayed. In that sense, the viewing ratecan be used as a ground for setting charges to display in a case wherethe different image is provided as advertisements.

FIG. 6 is a diagram explanatory of a timing for the different imagepresentation part 66 to present the different image in the region ofnon-interest. This example indicates the case in which, on the basis ofthe first condition above, an image of an object of which the temporalchanges in pixel values are equal to or smaller than a predeterminedvalue is taken as the region of non-interest in which the differentimage is to be displayed. Here, the changes in pixel values on thevertical axis of the graph are, for example, averages of the temporalchanges per unit time in the values of the pixels constituting the imageof the subject appearing in the moving image, or of the temporal changesper unit time in the pixel values of the sampling points in the image.

The region extraction part 62 acquires these changes in pixel values atpredetermined intervals over time along the horizontal axis of thegraph. A target of which the changes in pixel values are to be acquiredmay be narrowed down beforehand on the basis of the colors and positionsof images for example. If the acquired changes in pixel values do notexceed a threshold value Pth in a period of determination from time 0 toa predetermined time T1 in the graph, the region extraction part 62decides this image to be the region of non-interest.

According to this determination, the different image presentation part66 starts composing the different image in the region of non-interestfrom time T1 on. Meanwhile, the region extraction part 62 continuouslyacquires the changes in pixel values in the original moving image. Inthe illustrated example, the changes in pixel values in the region ofnon-interest start to increase from a certain time point and exceed thethreshold value Pth at time T2. For example, even the background such asthe sky that is not generally likely to draw attention may temporarilypresent considerable changes when fireworks are set off or laser beamsare displayed as a side show of an event. At this time, the object thathas not draw an attention is highly likely to draw attention all of asudden.

In view of the above, the region extraction part 62 also monitors thechanges in pixel values of a region which has been decided as a regionof non-interest to detect whether the region is no longer the region ofnon-interest. That is, the region extraction part 62 detects that thechanges in pixel values exceed the threshold value Pth at time T2, andthus eliminates the region as the region of non-interest. In response tothis, the different image presentation part 66 hides the different imagehaving been displayed so far in the region. This avoids the situationwhere the object of interest such as fireworks is hidden by thedifferent image.

Then, at time T3 at which the changes in pixel values become equal to orless than the threshold value Pth, the region extraction part 62 detectsthat the region has again become the region of non-interest. In responseto this, the different image presentation part 66 resumes composing thedifferent image in the region of non-interest. Note that the illustrateddisplay/non-display control of the different image can also beimplemented using the frequency at which the user's line of sight isdirected instead of the changes in pixel values. Also, the thresholdvalue at which the different image is switched from the display state tothe non-display state may be the same as, or different from, thethreshold value at which the different image is switched from thenon-display state to the display state.

Furthermore, multiple parameters may be used as the reference fordetermining the display or non-display of the different image. Forexample, whereas the region of non-interest at time T1 may be decided onthe basis of the changes in pixel values or the frequency at which theuser's line of sight is directed, during the periods T2 to T3 where thedifferent image is not displayed, the period in which the display of thedifferent image is not desired, such as the period during whichfireworks are scheduled to be set off is preliminarily identified andmanually set as non-display. As another alternative, the different imagemay not be displayed during the entire period illustrated in which thedifferent image is displayed. For example, during the display period,the different image may be displayed at a timing designated by theserver 200.

FIG. 7 is a diagram schematically depicting an example of the imagedisplayed in the present embodiment. The illustrated image is a frame ofa video image of a car race. Two different images 130 and 132 indicativeof an advertisement of canned products are composed in the frame. Of thetwo images, the different image 130 superposed on the image of themountains in the relatively foreground is displayed with a magnificationfactor larger than that of the different image 132 superposed on theimage of the sky in the background.

In this manner, the different images displayed with magnificationfactors corresponding to the actual distances of the respectivereplacement objects provide a sense of unity with the original image.Note that, in a case where the surface of a replacement object isinclined or curved relative to the camera and the distance varies withthe position over the same surface, the display magnification factor ofone different image composed on that surface may also be varied with theposition. This makes it possible artificially to produce a situation inwhich the different image remains pasted on the replacement object. Evenin such a case, however, the display magnification factor of onedifferent image is fixed regardless of the position, thereby making itpossible to promote the visibility.

What has been explained so far is how the different image is composed inthe region of non-interest in the moving image displayed on thehead-mounted display 100. Alternatively, a different image may bedisplayed on AR glasses letting real-world images pass through, in sucha manner that the different image is merged with the real image. In thiscase, too, the region extraction part 62 in the image processingapparatus 10 extracts the region of non-interest from a frame of themoving image concurrently presenting the space being viewed by the userthrough the AR glasses. The different image presentation part 66 thencauses the different image to be displayed in a region corresponding tothe region of non-interest in the real image viewed through the ARglasses, i.e., in the real-image region of the replacement object.

FIG. 8 is a diagram explanatory of how to obtain a viewing rate at atime of displaying a different image on AR glasses worn by audienceduring a concert in the real world. Subfigure (a) first depicts aconcert hall in the real world. It is assumed that the audience in seats140 wear AR glasses. In a state where no different image is displayed,the lines of sight of the audience are naturally focused on players on astage. In the drawing, some of the lines of sight of the audience arerepresented by arrows.

In Subfigure (b), on the other hand, the right and left walls of thehall are used as the replacement objects on which different images 142and 144 are displayed. That is, in the actual state of Subfigure (a),the audience wearing the AR glasses can see the different images 142 and144 apparently displayed on both walls. In such a state, there is agrowing possibility that the lines of sight of the audience are directedto the different images as indicated by arrows. The viewing ratecalculation part 56 then acquires changes in the lines of sight beforeand after the different images are displayed as described above.Specifically, the viewing rate calculation part 56 collects movements ofthe lines of sight of the audience from the image processing apparatus10 that causes the images to be displayed on the AR glasses worn by theaudience.

Thereafter the viewing rate calculation part 56 calculates an average,per spectator, of the accumulated times in which the lines of sight havebeen directed to the right and left walls over a predetermined timeperiod. For example, if the average values are assumed to be t1 and t2before and after display of the different images, respectively, theviewing rate calculation part 56 calculates t2/t1 or t2-t1 as theviewing rate. Alternatively, the number of times the lines of sight havebeen directed may be used in place of the accumulated times in which thelines of sight have been directed. Since the higher the viewing rate,the higher the effects of the different images being displayed areassumed to be, the viewing rate can be used as the ground for settingadvertisement charges and for optimizing the replacement objects onwhich the different images are to be displayed, for example.

Note that similar viewing rates can also be acquired in a mode in whicha different image is composed in the moving image displayed on thehead-mounted display 100. That is, in a situation where the moving imageof the concert is displayed as depicted in Subfigure (a), the imageprocessing apparatus 10 composes the different images 142 and 144 asillustrated in Subfigure (b). In this case, too, it is possible toobtain changes in the line of sight of the user wearing the head-mounteddisplay 100. The viewing rate calculation part 56 in the server 200 cancalculate the viewing rate in a manner similar to what has beendiscussed above by collecting the changes in the lines of sight ofmultiple users.

Explained next are the workings of the image processing apparatus 10implemented by use of the above-described configuration. FIG. 9 is aflowchart indicative of processing steps for displaying a content image,the steps being performed by the image processing apparatus 10 in thepresent embodiment. The steps of this flowchart are started when theuser wearing the head-mounted display 100 performs input on the imageprocessing apparatus 10 to select the content desired to be viewed.First, the image acquisition part 60 in the image processing apparatus10 starts acquiring moving image data of the selected content (S10).

The data may be transmitted from the server 200 or retained inside theimage processing apparatus 10 as discussed above. On the basis ofpredetermined conditions, the region extraction part 62 extracts theregion of non-interest from the moving image (S12). The informationnecessary for the extraction may be acquired from the server 200 orretained inside the image processing apparatus 10. As anotheralternative, the position information acquisition part 64 may acquirethe position information regarding the subject and use the acquiredinformation for extracting the region of non-interest.

Depending on the extraction condition, it may take at least a certainamount of time to do the extraction. For this reason, the output part 70may output the moving image data to the head-mounted display 100concurrently with the process in S12. Immediately after the extraction,the changes in pixel values of the region of non-interest and thefrequency at which the line of sight is directed are obviously equal toor less than their respective threshold values (Y in S14). Thus, thedifferent image presentation part 66 displays the different image in theextracted region of non-interest (S16).

At this point, the different image presentation part 66 may decide thedisplay magnification factor of the different image according to thedistance of the replacement object acquired by the position informationacquisition part 64. The data of the different image may be read outfrom the different image data storage part 68 or acquired from theserver 200 when needed. The output part 70 outputs to the head-mounteddisplay 100 the data of the display image in which the different imageis composed (S18). If there is no need to terminate the display (N inS20), the different image is continuously output and caused to bedisplayed in subsequent frames (Yin S14, S16, and S18).

However, if the region extraction part 62 detects that the changes inpixel values of the region of non-interest or the frequency at which theline of sight is directed has exceeded the corresponding threshold value(N in S14), the process in S16 is skipped in order to place thedifferent image in a non-display state. It is to be noted that thedetermination in S14 is not necessarily performed in units of a frame.Preferably, a predetermined time period for the determination may beestablished, so that the display state and the non-display state of thedifferent image are not switched with one another at short timeintervals. If the moving image is ended or if the display needs to beterminated as a result of the user's stop operation, the wholeprocessing is brought to an end (Y in S20).

According to the above-described embodiments of the present invention, aregion of an image of a target object not included in the object ofinterest in the moving image of content is extracted as the region ofnon-interest. An image different from the target object is thendisplayed in the extracted region. This promotes effective utilizationof the screen based on the configuration of the original image andthereby presents diverse pieces of information at the same time. When anactually presented image is used as the destination in which thedifferent image is to be displayed, a sense of unity is brought aboutfor the entire image. As a result, the user can recognize the presentedinformation naturally without shifting his or her line of sightconsiderably from the moving image.

Also, whether or not to display the different image and how to determinethe magnification factor of the different image are decided on the basisof the actual circumstances such as the positions of the target objects,the manner in which images are displayed, and the frequency at which theuser's line of sight is directed. This enables the different image to bedisplayed comfortably while respecting the details of the moving imageand the intentions of the user. Particularly in image representationusing a head-mounted display, the inventive display does not impair therealistic sensations as compared to simply displaying information in afixed region on the screen. Also, the amount of the informationconstituting the image of the original replacement object is reducedbeforehand, so that it is also possible to reduce the size of the datato be transmitted from the server.

The present invention has been described above in conjunction withspecific embodiments. It is to be understood by those skilled in the artthat suitable combinations of the constituent elements and of variousprocesses of the embodiments described above as examples will lead tofurther variations of the present invention and that such variationsalso fall within the scope of this invention.

For example, whereas the image processing apparatus 10 in the presentembodiment composes the different image immediately before displayingit, similar effects are obtained when the server 200 composes thedifferent image and distributes it to the image processing apparatus 10.In this case, the server 200 is configured with the image acquisitionpart 60, the region extraction part 62, and the different imagepresentation part 66. The distribution processing part 50 need onlytransmit to the image processing apparatus 10 the data of the movingimage in which the different image is composed.

INDUSTRIAL APPLICABILITY

As discussed above, the present invention may be applied to diversekinds of information processing apparatuses including an imageprocessing apparatus, a wearable display, a game console, a mobileterminal, a personal computer, and an image distribution server, as wellas to systems including any of these apparatuses.

REFERENCE SIGNS LIST

-   -   10 Image processing apparatus    -   23 CPU    -   24 GPU    -   26 Main memory    -   32 Communication part    -   34 Storage part    -   36 Output part    -   38 Input part    -   40 Recording medium driving part    -   50 Distribution processing part    -   52 Content data storage part    -   54 Region information acquisition part    -   56 Viewing rate calculation part    -   60 Image acquisition part    -   62 Region extraction part    -   64 Position information acquisition part    -   66 Different image presentation part    -   68 Different image data storage part    -   70 Output part    -   72 Line-of-sight information acquisition part    -   100 Head-mounted display    -   110 Stereo camera    -   200 Server

The invention claimed is:
 1. An image distribution system comprising: animage processing apparatus; and a server, wherein the image processingapparatus includes an image acquisition part configured to acquire dataof a moving image representing a space to be viewed, a region extractionpart configured to extract, as a region of non-interest, a region of animage of any one of target objects not included in an object of interestin a frame of the moving image, and a different image presentation partconfigured to present and output an image different from the image ofthe target object in the region of non-interest, wherein the servertransmits the data of the moving image to the image processingapparatus, the server including a viewing rate calculation partconfigured to acquire information regarding a direction of a line ofsight of a user viewing an image output from the image processingapparatus, the viewing rate calculation part further calculating aviewing rate of the different image on a basis of changes in thedirection of the line of sight before and after the different image isdisplayed, and wherein at least one of: (i) the region extraction partdetects a time period in which the temporal changes in the pixel valuesof the constituent pixels constituting the region of non-interest exceeda predetermined value, and (ii) the image acquisition part acquires thedata of the moving image and information regarding a condition forextracting the region of non-interest from a server over a network. 2.The image distribution system according to claim 1, wherein the regionextraction part extracts, as the region of non-interest, the region ofwhich the constituent pixels have temporal changes in pixel values thatare equal to or less than a predetermined value.
 3. The imagedistribution system according to claim 1, wherein the image acquisitionpart acquires registered information designating the target object, andthe region extraction part extracts the region of non-interestidentified by referencing the registered information.
 4. The imagedistribution system according to claim 1, further comprising: aline-of-sight information acquisition part configured to acquireinformation regarding a direction of a line of sight of a user, whereinthe region extraction part extracts, as the region of non-interest, aregion of an image to which the line of sight is directed at a frequencyequal to or less than a predetermined value.
 5. The image distributionsystem according to claim 1, wherein the region extraction part extractsthe region of an image of the sky or of a ceiling as the target object.6. The image distribution system according to claim 1, wherein, given anupper region and a lower region vertically bisecting the frame, theregion extraction part extracts the region of non-interest from theupper region.
 7. The image distribution system according to claim 1,further comprising: a position information acquisition part configuredto acquire position information regarding a subject in athree-dimensional image capturing space, wherein the region extractionpart extracts, as the region of non-interest, a region of an image ofthe subject of which a distance from a camera capturing the moving imageis equal to or longer than a predetermined value.
 8. The imagedistribution system according to claim 1, wherein the different imagepresentation part presents the different image with a magnificationfactor corresponding to the distance of the target object appearing inthe region of non-interest from the camera capturing the moving image.9. The image distribution system according to claim 1, wherein thedifferent image presentation part places the different image in anon-display state during the time period.
 10. The image distributionsystem according to claim 1, wherein the image acquisition partacquires, from a server over a network, the data of the moving image inwhich an amount of information constituting the image of the targetobject in the region of non-interest out of the frame of the movingimage is reduced.
 11. The image distribution system according to claim1, wherein the region extraction part extracts the region ofnon-interest on a basis of the information regarding the condition. 12.The image distribution system according to claim 1, wherein thedifferent image presentation part presents the different image in aregion corresponding to the region of non-interest within a field ofview of augmented reality glasses worn by a user viewing the spacerepresented by the moving image.
 13. An image distribution systemcomprising: a server; and an image processing apparatus, wherein theserver includes an image acquisition part configured to acquire data ofa moving image representing a space to be viewed, a region extractionpart configured to extract, as a region of non-interest, a region of animage of any one of target objects not included in an object of interestin a frame of the moving image, and a different image presentation partconfigured to present an image different from the image of the targetobject in the region of non-interest, wherein the image processingapparatus acquires over a network the data of the moving image outputfrom the server and representing the different image, the imageprocessing apparatus further causing a display apparatus to display themoving image, wherein the server acquires information regarding adirection of a line of sight of a user viewing an image output from theimage processing apparatus, the server further calculating a viewingrate of the different image on a basis of changes in the direction ofthe line of sight before and after the different image is displayed, andwherein at least one of: (i) the region extraction part detects a timeperiod in which the temporal changes in the pixel values of theconstituent pixels constituting the region of non-interest exceed apredetermined value, and (ii) the image acquisition part acquires thedata of the moving image and information regarding a condition forextracting the region of non-interest from a server over a network. 14.An image distribution method comprising: acquiring data of a movingimage representing a space to be viewed; extracting, as a region ofnon-interest, a region of an image of any one of target objects notincluded in an object of interest in a frame of the moving image;presenting and outputting an image different from the image of thetarget object in the region of non-interest; causing a server totransmit the data of the moving image to the image processing apparatus,the server being further caused to acquire information regarding adirection of a line of sight of a user viewing an image output from theimage processing apparatus, the server being further caused to calculatea viewing rate of the different image on a basis of changes in thedirection of the line of sight before and after the different image isdisplayed; and at least one of: (i) detecting a time period in which thetemporal changes in the pixel values of the constituent pixelsconstituting the region of non-interest exceed a predetermined value,and (ii) acquiring the data of the moving image and informationregarding a condition for extracting the region of non-interest from aserver over a network.